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Executive  Summary 


The  Naval  Postgraduate  School  was  contracted  by  NAVNETWARCOM  /  SPAWAR 
PEO  C4I  to  develop  objectives  and  metrics  for  MDA  evaluation,  to  include  Spiral- 1 
system  assessments  and  follow-on  full  operational  capabilities  assessments. 

NPS  applied  a  robust  method  for  military  testing/experimentation.  That  method, 
previously  developed  for  Trident  Warrior  and  other  assessment  events  applies: 

•  A  structure  for  defining  program  objectives  and  goals. 

•  A  structure  for  defining  metrics. 

•  Methods  of  specifying  data  requirements  for  operational  and  tactical  events. 

•  Methods  of  rapidly  generating  reports  from  collected  data  on  metrics  pertinent  to 
program  objectives  and  goals 

The  FORCEnet  Innovation  and  Research  Enterprise  (FIRE)  knowledge  management 
system  provides  robust  support  for  applying  this  method  and  managing  the  knowledge  it 
produces.  In  particular,  FIRE  provides  forms  for  planning  and  reporting  within  and  across 
events.  It  also  provides  work  and  collaboration  spaces. 

NPS  applied  this  method  and  infrastructure  to  develop  an  MDA-specific  structure  of 
objectives  and  metrics.  Authoritative  MDA  documents  served  as  the  foundation  for  this 
work.  This  status  report: 

(1)  Describes  the  NPS  method  for  developing  objectives  and  metrics  and  their  use  in 
test  planning. 

(2)  Defines  the  framework  NPS  has  developed  for  MDA  assessment. 

(3)  Presents  objectives  and  metrics  that  have  been  defined  from  MDA  program 
requirements  documentation. 

(4)  Describes  FIRE  forms  and  reports  that  have  been  created  for  MDA  program  use. 

The  NPS  products  for  MDA  will  support  the  full  range  of  required  MDA  capability 
testing  for  Spirals  1,  2,  and  beyond,  including: 

•  System  capabilities 

•  Systems  support  for  operations 

•  Operations  process  capabilities 

•  Workflow  and  information  flow 

•  Guidance  documents  status 

•  Organization  and  cross-organization  capabilities 

•  Cross-organization  and  multi-national  agreements  and  processes 

•  Human  capabilities 

Further,  this  work  will  enable  the  Navy  to  correlate  and  fuse  findings  across  many 
dimensions: 
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•  Across  test  venues 

•  Across  studies  of  process,  system,  organization,  human,  and  guidance 

•  Across  experiments,  e.g.,  Trident  Warrior,  Empire  Challenge,  and  JMMES  JCTD. 

The  work  reported  here  is  in  process.  NPS  has  developed  an  MDA-specilic  framework  of 
objectives  and  metrics.  Further  effort  is  required  to  develop  measures  within  this 
framework,  apply  them  at  evaluation  events,  and  report  results  within  events  and  across 
the  MDA  program.  NPS  recommends  that  the  development  of  measures  continue  with  a 
specific  focus  on  generating  measures  for  specific  events. 
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1.  INTRODUCTION 


The  Naval  Postgraduate  School  was  contracted  by  NAVNETWARCOM  /  SPAWAR 
PEO  C4I  to  develop  objectives  and  metrics  for  MDA  evaluation,  to  include  Spiral- 1 
system  assessments  and  follow-on  full  operational  capabilities  assessments.  This  report: 

(5)  Describes  the  NPS  method  for  developing  objectives  and  metrics  and  their  use  in 
test  planning. 

(6)  Defines  the  framework  NPS  has  developed  for  MDA  assessment. 

(7)  Presents  objectives  and  metrics  that  have  been  defined  from  MDA  program 
requirements  documentation. 

(8)  Describes  FIRE  forms  and  reports  that  have  been  created  for  MDA  program  use. 

The  NPS  effort  focuses  on  defining  test  objectives  and  their  associated  metrics  to 
evaluate: 

•  The  operational  utility  and  impact  of  Spiral  1  technologies 

•  MDA  process  capabilities 

•  Human  and  organization  capabilities 

•  Quality  of  MDA  CONOPS  and  TTP  documents 

•  Multi-organization  and  multi-national  agreements 

This  effort  builds  on  an  established  method  for  experiment/test  planning1,  and  it 
leverages  a  special-purpose  knowledge  management  system  -  FORCEnet  Innovation  and 
Research  Enterprise  (FIRE)  -  that  supports  experiment/test  planning,  results 
development,  and  reporting. 

Further,  the  NPS  product  builds  on  an  authoritative  set  of  MDA  resources: 

•  Campaign  Plan  for  Navy  Maritime  Domain  Awareness  Prototypes,  21  Aug  200 

•  MDA  Spiral  1  Overarching  T  &  E  Plan,  1  Oct  2007 

•  Scoping  Document,  version  4-4,  23  Jan  2008 

•  Fleet  MDA  CONOPS,  1 3  Mar  2007 

•  MDA  Focus  Area  Brief,  ONR  S&T  Goals  presentation,  Jul  2007 


1  This  process  is  documented  in  “FIRE  Experiment  Planning  and  Reporting  Structure”,  NPS-IS-07-002,  Jul 
2007.  A  second  report  describes  how  the  products  of  this  process  are  mapped  to  JCIDS  or  other  areas  of 
interest:  “Mapping  Experimental  Results  to  Operational  Capabilities,”  NPS-97-08-001,  Nov  2007. 
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•  The  MDA  Workflow  developed  by  NPS  (see  the  accompanying  report)  and  vetted 
by  representatives  of  ASN  RDA,  C3F,  COTF,  Dept,  of  the  Under  Secretary  of  the 
Navy,  DISA,  HFE  LLC,  JITIC,  METRON,  MIFCLANT,  MIFCPAC, 

NAVCENT,  NAVNETWARCOM,  NCIS,  NORTHCOM,  NPS,  NRL,  NWDC, 
ONE  OPNAV,  PMW  120,  SPAWAR,  and  MDA  Spiral  1  technology  experts. 

The  readers  should  be  aware  that  this  reports  documents  one  of  several  metrics  and 
measurement  efforts  for  MDA.  Those  efforts,  in  combination  with  the  NPS  work,  define 
a  broad  space  of  MDA  metrics.  The  reader  may  wish  to  access  these  additional  efforts  if 
their  interests  extend  beyond  the  measurement  space  NPS  is  addressing.  Specifically: 

•  Measures  developed  by  Mike  Shumberger  of  HFE  for  the  Navy  focus  on 
assessment  of  fleet  readiness  for  MDA  operations. 

•  Measures  developed  by  Fifth  Fleet  support  acquisition  and  use  decisions  by 
operational  organizations  concerning  existing  operations. 
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2.  FOUNDATIONS 

The  MDA  products  described  below  are  built  on  a  foundation  that  has  two  dimensions: 
objectives  and  metrics.  We  define  those  terms  below,  and  then  describe  how  they  are 
instantiated  for  MDA. 

2.1  Objectives 

Objectives  define  what  is  to  be  learned  from  an  investigation.  For  MDA  these  objectives 
are  to  assess  current  capabilities  or  to  develop  new  ones.  A  logical  objective  structure 
enables  us  to  correlate  test  results  from  different  venues  and  to  relate  them  to  other  areas, 
such  MHQ  w/MOC.  For  the  MDA  effort,  NPS  developed  an  objective  structure  that 
consists  of  two  substructures:  “program”  and  “test”  objectives. 

Program  Substructure 

Program  area:  A  major  area  of  interest  to  the  MDA  program.  Example:  Detect 
and  Track 

Activity  /  Focus:  An  element  or  activity  within  a  program  area.  Example: 
Ship  Detection 

Program  objective:  A  focal  element  or  activity  within  the  MDA 
activity,  Example:  Provide  non-radiating  ship  detection. 

The  above  is  the  “program”  portion  of  the  structure.  It  applies  to  the  program  as  a  whole 
and  contains  all  of  the  objectives,  implied  or  stated,  from  the  documents  listed  in  Section 
1 .  The  Program  Objectives  for  MDA  have  been  input  in  FIRE  so  that  they  can  be 
selected  and  used,  as  needed,  for  specific  test  venues. 

Test  Objectives  Substructure 

Test  objective:  A  focus  of  one  or  more  test  venues.  Example:  Provide  automated 
detection  of  non-radiating  ships. 

Objective-  Goal(s):  A  determination  to  be  made  or  question  to  be 
answered  at  a  test  venue.  Note  that  objective  goals  are  operationalized  by 
(consist  of)  situation,  specific  measures  and  data  (measurements). 
Example:  Determine  if  Global  Hawk  provides  accurate  and  timely 
detection  of  non-radiating  ships. 

The  “test”  portion  of  the  structure  specifies  the  objectives  and  goals  to  be  determined  for 
a  particular  test  venue. 

Designing  a  test  involves  several  more  components  in  addition  to  the  objectives  and 
goals.  One  also  has  to  define: 

•  Specific  measures 

•  Data  required  to  produce  those  measures 

•  Situations  to  be  set  up  so  that  the  correct  data  are  captured 

•  A  schedule  that  combines  these  components 
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2.2  Metrics 

Metrics  are  the  attributes,  measures,  and  standards  associated  with  an  activity,  such  as 
ISR  or  support  activity  such  as  network-centric  operations. 

•  Attributes  are  single -word  expressions  of  the  characteristics  of  people,  things,  or 
processes.  Example:  Timely 

•  Measures  quantify  attributes.  Example:  Average  time  from  submission  of  RFI  to 
receipt  of  requested  information.  Measures  are  of  several  types: 

o  Measures  of  Effectiveness  (MOEs)  concern  how  well  a  technology, 
organization,  or  process  performs  its  functions.  Example:  Reliable 
o  Measures  of  Performance  (MOPs)  concern  a  specific  parameter  of  a 
technology,  organization,  or  process.  Example  for  “reliable”  effects: 
Robust 

o  Measures  of  Utility  (MOUs)  concern  how  well  a  technology,  organization, 
or  process  contributes  to  a  military  activity.  Example:  Needed 
o  Measures  of  Readiness  (MORs)  reflect  the  combined  effectiveness  and 
utility  of  a  system,  and  its  life-cycle  plan. 

•  Standards  are  values  that  specify  a  satisfactory  performance  boundary.  Example: 
Two  hours  from  submission  of  RFI  to  receipt  of  requested  information. 

Attributes  and  measures  have  meaning  only  when  associated  with  a  task.  For  example: 

•  Task:  retrieval  of  data  from  a  local  repository 

•  Attributes:  timely 

•  Measure:  latency  (or  delay)  of  retrieved  infonnation. 

In  this  case,  the  task  helps  to  specify  the  meaning  of  timeliness.  The  actual  measure  of 
interest  completes  the  specification.  This  association  aspect  will  be  covered  more 
completely  in  Appendix  D. 

This  complete  structure,  objectives,  goals,  metrics,  and  other  planning  components  that 
define  events  and  data,  are  provided  in  the  FIRE  planning  and  reporting  system.  The 
system  contains  input/edit  forms  and  reports  that  show  current  contents  of  the  test 
database.  FIRE  is  described  in  Section  5. 
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3.  MDA  Use-Case:  Scenario  and  Issues 

MDA  objectives  are  complex  and  multi-threaded.  We  might,  for  example,  want  to 
evaluate  Vol  (Vessel  of  Interest)  handoff  processes,  the  systems  that  support  the  process, 
and  the  associated  TTP.  In  the  remainder  of  this  report,  we  describe  how  we  put  the 
structures  above,  and  FIRE,  to  work  to  define  objectives  and  metrics  in  the  MDA 
program.  We  illustrate  this  with  a  use  case,  which  in  essence  describes  the  military 
operations  that  will  be  invoked  in  a  study  and  what  we  wish  to  learn  from  studying  them. 
Once  this  has  been  established,  we  use  the  MDA  objectives  and  metrics  structure  to 
define  the  specifics  of  the  test(s)  to  be  undertaken. 

In  this  use-case  the  following  operations  take  place: 

•  HUMINT  identification  of  a  person-of-interest. 

•  Ship  cargo  and  personnel  identification. 

•  Non-AIS  and  AIS  ship  tracking. 

•  Information  processing  and  sharing  for  Situation  Awareness. 

•  Vessel-of-Interest  handoff  across  AORs  and  nations. 

•  Vessel  boarding. 

The  MDA  issues  to  be  investigated  are: 

•  Database  access. 

•  System  interoperability. 

•  Multi-national  and  multi-agency  cooperation  agreements. 

•  CONOPS  and  TTP  sufficiency. 

•  Information  sharing  agreements  and  information  interoperability. 

•  Information  quality  for  SA  and  decision-making. 

•  Boarding  communications  and  surveillance  capabilities. 

•  Boarding  data  and  information  collection. 

As  noted  above,  the  use  case  and  issues  identify  learning  objectives.  An  issue  can  apply 
to  more  than  one  operation  and  an  operation  can  address  several  issues.  Table  1  provides 
a  partial  listing  of  the  issues  to  be  addresses  for  each  operation. 
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Operations 

Issues  (notional,  not  complete) 

HUMINT 

Access  to  injected  HUMINT  information. 

Timeliness  of  HUMINT  alert. 

Access  to  human  information  databases. 

Ship 

Access  to  ship  manifests. 

Information 

Ship  database  access. 

Completeness  of  ship  information 

Ship 

Tracking 

Timeliness  and  accuracy  of  AIS  reports 

Timeliness  and  accuracy  of  overhead  surveillance. 

Timeliness  of  identification  process. 

Automated  reporting  capabilities. 

Vol  Handoff 

CONOPS  and  TTP  sufficiency. 

Personnel  familiarity  with  handoff  process. 

Timeliness  of  handoff  completion. 

Sufficiency  and  accuracy  of  handoff  information. 
Automated  M2M  handoff  capabilities. 

Information 

GUI  usefulness/clarity. 

Processing 

Ability  to  correlate  multi-int  information. 

Reachback  timeliness  and  sufficiency. 

Ability  to  correlate  HUMINT  with  person  database. 

Information  fusion  accuracy. 

Information 

Multi-national  agreements,  sufficiency/constraints. 

Sharing 

Multi-agency  agreements,  sufficiency/constraints. 
M2M  interoperability. 

Content  and  format  interoperability. 

Distributed  information  fidelity. 

Boarding 

Available  communications  bandwidth/throughput. 
Communications  security. 

Communications  reliability. 

Biometrics  kit  usability. 

Real-time  assessment  capabilities. 

Table  1.  MDA  operations  and  issues  for  an  example  use-case. 


Note:  Some  issues  in  Table  1  are  highlighted  in  yellow.  They  are  addressed  in  Section  6 
below. 
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We  leverage  this  use  case  in  the  remainder  of  this  report.  Specifically,  we: 

•  Describe  the  objectives  and  metrics  structure  (Section  4). 

•  Describe  use  of  FIRE  for  test  planning  and  reporting  (Section  5). 

•  Address  the  use-case  as  an  illustration  of  test  planning  (Section  6). 
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4.  MDA  Objectives  and  Metrics 

The  following  is  the  MDA  program  structure,  a  specific  instantiation  of  the  objectives 
and  metrics  structure  described  above.  In  Section  6,  we  describe  application  of  this 
structure  to  specific  tests.  Appendix  E  describes  how  this  structure  correlates  with  the 
JCIDS  JCAs. 

4.1  MDA  Objectives 

Eight  MDA  Program  Areas  have  been  defined: 

•  Detect/Track  -  detection  and  tracking  of  surface  vessels  by  any  means 

•  Process  -  processing  of  data  and  information 

•  Analyze/Develop  SA  -  analysis  of  processed  ship  information  to  provide  threat 
assessment  and  develop  situation  awareness 

•  Distribute/Share  -  distribution  and  sharing  of  assessments  for  course-of-action 
development  and  decision-making. 

•  Archive/Retrieve  -  deploying,  maintaining,  and  updating  MDA  databases 

•  Guidance  -  assessment  of  guidance  quality,  CONOPS,  TTP,  directives 

•  Workflow  -  workflow  assessment,  including  the  influences  of  humans, 
organizations,  and  workflow  structure 

In  addition,  it  was  detennined  that  E-MIO  operations  should  be  broken  out  as  its  own 
program  area  from  other  operations. 

•  E-MIO  -  operations  capability:  planning,  execution,  and  information 
management  activities  associated  with  E-MIO 

All  of  the  MDA  objectives  in  the  reference  documents  fit  within  these  Areas. 

Within  each  Program  Area,  several  Activities  or  test  foci  have  been  defined.  These  are 
presented  in  Table  2. 
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Program  Area 

Activity  /  Focus 

Detect/T  rack 

Ship  Detection 
Ship  Tracking 

Acquire  Database  Information 
Intelligence  Collection 

Analyze/Develop  SA 

Develop  Profiles 
Request  Information 
Classify  Vessels 
Prioritize  Information 

Archive/Retrieve 

Deploy  Repository 
Acquire  Information 
Authenticate  Information 
Manage  Access 
Assure  Information 


Workflow 

Task  Assignments 

Organization 

Group 

Human 

Multi-Organization  Workflow 


Program  Area 

Activity  /  Focus 

Process 

Identify  Ship's  Data 

Classify 

Correlate 

Fuse 

Distribute/Share 

Distribution  Means 

Collaborate 

Disseminate 

Guidance 

TTP  &  SOP 
CONOPS 
Standing  Orders 
Cross-Organization  Agreements 

E-MIO  Operations 

Information  Acquisition 
Develop  Situation  Understanding 
Course-of-Action 
Boarding  Execution 
Shipboard  Collection 
Information  Dissemination 
Mission  Assessment 


Table  2.  First  two  levels  of  the  MDA  program  objectives  structure. 


Many  of  the  processes  in  E-MIO  are  included  in  the  other  Program  Areas.  However,  E- 
MIO  is  broken  out  as  its  own  Program  Area  at  the  request  of  those  evaluating  MDA.  This 
could  be  accomplished  with  other  operations,  if  desired. 
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Within  each  Activity  /  Focus  several  Program  Objectives  have  been  developed.  Table  3 
presents  examples  for  the  Detect/Track  Program  Area.  Also  shown  is  the  coding  used 
(e.g.,  DT-ST.  1),  which  is  explained  below.  The  full  list  is  presented  in  Appendix  D. 


Program  Area 

Activity  /  Focus 

Program  Objective 

DT  Detect/Track 

DT-SD  Ship  Detection 

DT-SD.l 

Non-radiating-ship  detection 

DT-SD.2 

Radiating-ship  detection 

DT-SD. 3 

Plan  &  Optimize  Collection 

DT-SD.4 

Automated  alerting 

DT-ST  Ship  Tracking 

DT-ST.  1 

Non-radiating-ship  tracking 

DT-ST.2 

Radiating-ship  tracking 

DT-ST.3 

AlS-broadcasting-ship  tracking 

DT-ST.4 

Handoff 

DT-ST.5 

All  ship  tracking 

DT-Acq 

Acquire  Database  Information 

DT-Acq.  1 

Local  repository  retrieval 

DT-Acq  .2 

Remote  repository  retrieval 

DT-Acq.  3 

Cross-domain  retrieval 

DT-Acq  .4 

Weather  &  Environment 

DT-Acq.  5 

Reachback/RFI 

Program  Objectives  are  written  in 
shorthand,  with  the  word  “Provide” 
removed.  E.g.,  DT-SD.l  is  “Provide 
non-radiating-ship  detection. 


Color  is  used  in  the  table  only  to 
show  that  the  Program  Objectives 
belong  to  the  same  Activity/Focus. 


Table  3.  Example  Activity  Types  for  the  Monitoring  &  Collection  Activity  Category. 

4.2  MDA  Metrics 

As  explained  above,  metrics  define  the  attributes,  measures,  and  standards  associated 
with  an  activity,  such  as  a  network-centric  operational  or  support  activity.  The  measures 
are  enumerated  below  through  their  attribute  pair.  Appendix  A  contains  detailed 
definition  of  each  attribute.  NPS  developed  the  attribute  structure  for 
NAVNETWARCOM,  which  uses  it  for  their  CBAs.  NPS  added  Readiness  and  its 
components  for  the  MDA  program. 

Four  MOEs  form  the  basis  for  the  structure.  They  are: 

•  Accessible  You  can  get  to  it. 

•  Reliable  It  is  there  when  needed. 
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•  Capable  It/he/she/they  can  do  the  defined  job. 

•  Usable  You  can  use  it. 

Each  MOE  has  a  set  of  included  MOP.  These  21  MOPs  are  shown  in  Table  4. 


Effective 

Accessible 

Reliable  Capable 

Usable 

MOE 

Capacity 

Available 

Compatible 

Extensive 

Efficient 

Robust 

Persistent 

Secure 

Assured 

Sufficient 

Flexible 

Accurate 

Timely 

Reach 

Automatic 

Clear 

Trusted 

Manageable 

Relevant 

Compliant 

Deployable 

MOP 

ii 

ii 

ii 

ii 

n 

Military  Utility 

Improved 


Needed 


Applicable 


Wanted 


Effective 

Utility 

Life  Cycle 

MOU 


Personnel  mor 


Table  4.  Attribute  structures  for  effectiveness,  utility,  and  readiness. 


Four  MOUs  were  defined: 

•  Improved  Improves  the  performance  of  operational  activities. 


•  Needed 

•  Applicable 


Fills  a  gap  in  current  capabilities. 

Can  be  applied  to  activity  perfonnance. 

•  Wanted  Operational  personnel  want,  will  use,  the  capability. 

Note  that  no  MOP  equivalents  were  defined  for  Military  Utility.  This  is  because 
currently  most  utility  determinations  are  subjective.  Objective  determinations  can  be 
made,  e.g.,  the  number  of  times  a  capability  is  used  as  a  measure  for  Wanted.  The  MOPs 
shown  for  effectiveness  are  also  appropriate  for  utility. 


In  addition,  a  readiness  metric  was  defined. 

•  Ready  Ready  is  an  official  procurement  term  that  refers  to  the  system 

being  ready 

for  fielding.  As  indicated,  it  is  a  roll-up  of  the  other  fundamental 
measures  and  the  life-cycle  plan  (which  includes  a  personnel  plan). 


13 


5.  MDA  in  the  FIRE  Knowledge  Management  System 

NPS  has  developed  the  FORCEnet  Innovation  and  Research  Enterprise  (FIRE)  to  provide 
complete  support  for  experiment  and  test  planning  and  reporting.  An  MDA  section  has 
been  set  up  in  FIRE,  implementing  the  structure  described  above.  In  this  chapter,  we 
describe  the  system’s  capabilities  and  how  it  is  used. 

It  is  important  to  recognize  that  the  objective  statements  in  FIRE  are  Test  Objectives. 
They  are  the  actual  objectives  to  be  achieved  and  their  status  addressed  in  a  test  venue 

5.1  FIRE  Structure 

FIRE  has  two  basic  sections,  Planning  Forms  and  Workspace: 


Planning  Forms 

•  Detailed  planning  is  accomplished  through  entries  in  pre-set  forms. 

•  There  are  three  sets  of  planning  forms 

o  Objective 
o  Data/Events 
o  Results 

•  Each  forms  set  includes 

o  Input/Edit  fonn 

o  Report  that  shows  the  current  planning  database  content 

Workspace 

•  Contains  collaboration  capabilities  and  folders  for  information  to  be  shared 

•  Pre-created  folders  and  user  defined  folders 

•  Check- in/check-out  library  for  document  version  control 


Following  is  a  depiction  of  the  principal  components  of  the  planning  forms. 


Data/Event  Planning 

Events 

Situations 

Data  Specifics 

Collection  Procedures 

Collector  Instructions 

Archiving  Procedures 

Thread  Results 

Measures  Results 


Survey  Answers 


Test  Context 


Obj-Goal  Status 


Context  Impact 


Figure  1.  Principal  components  of  the  MDA  planning  forms  in  FIRE. 
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To  plan  a  test  (see 

Figure  2),  we  use  both  the  forms  (left)  and  the  workspace  (right)  in  a  sequence  such  as 
the  one  illustrated  here  (center). 


Figure  2.  Schematic  of  FIRE  forms  and  workspace  use. 
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5.2  FIRE  Forms  Use 


The  MDA  objective  planning  forms  in  FIRE  have  been  installed  with  the  following 
information  pre-installed  in  the  objective  planning  forms.  This  has  been  done  for  each 
Mission  Area. 

•  Test  Objectives 

•  Objective-Goals 

•  Goal  Attributes 

•  Measures  for  each  Goal  Attribute 

•  Survey  questions  for  each  Goal 

It  is  expected  that  the  pre-installed  planning  entries  will  need  some  modification.  Those 
entries  are  for  the  MDA  program  in  general  and  will  need  to  be  made  specific  for  the 
particular  test  venue.  For  example,  an  objective  and  objective-goal  for  ship  tracking  is 
written  system-agnostic.  That  is,  we  have  no  prior  system-specific  expectations  in  our 
assessment  methodology.  The  particular  system  being  tested  in  the  venue  will  need  to  be 
inserted  in  the  goal  statement. 

The  modification  process  proceeds  as  follows  for  each  test  to  be  done: 

•  Choose  the  Test  Objective  to  be  investigated,  e.g.,  a  ship  tracking  objective. 

•  Chose  the  Objective-Goal  under  that  Objective,  e.g.,  system  interoperability  for 
handoff. 

•  Modify  the  goal  statement  for 

o  Systems  to  be  tested 

o  Particular  attribute  to  be  determined,  e.g.,  timely,  accurate,  sufficient 

•  Specify  the  exact  measure  to  be  detennined,  e.g.,  beginning  and  end  points  for 
timeliness 

•  Specify  the  data  source 

•  Specify  any  survey  questions,  e.g.,  was  information  handoff  accurate. . . 

The  above  steps  are  done  using  the  Objective  Planning  forms.  Details  of  events  and  data 
capture  are  specified  using  the  Data/Event  Planning  fonns. 

Some  of  these  steps  will  be  illustrated  for  the  use-case  in  Section  6. 

As  noted  above,  the  FIRE  forms  include  reports  of  what  is  contained  in  the  database. 
There  is  a  report  for: 

•  Each  Objective-Goal 

•  Each  of  the  three  forms 

•  Each  Venue 

There  are  not  separate  FIRE  databases  for  each  venue.  As  test  venues  are  added,  its 
reports  are  placed  above  those  for  preceding  venues,  including  a  visual  delineation. 
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6.  MDA  Use  Case:  Test  Development 

The  MDA  framework  defined  above  is  used  to  develop  test  objectives,  goals,  and 
metrics.  Here,  we  define  several  MDA  program  objectives  that  bear  on  the  use  case 
defined  in  Section  2.  Five  issues  to  be  addressed  are  highlighted  in  yellow  in  Table  1. 
Appendix  D  contains  the  complete  list  of  MDA  Program  Objectives  and  associated 
attributes,  with  an  objective  code  which  is  used  here. 

6.1  Issues  Addressed  and  Program  Objectives 

Table  5  shows  the  issues  addressed  here,  again  highlighted  in  yellow.  The  table  also 
includes  the  corresponding  MDA  Program  Objectives  and  their  codes. 


Ship  Tracking  Code  Program  Objective 

Timeliness  and  accuracy  of  overhead  surveillance 

DT-ST.1  Non-radiating-ship  tracking 

DT-ST.2  Radiating-ship  tracking 

Vol  Handoff 

CONOPS  and  TTP  sufficiency 

G-TTP.4  Operational/Tactical  Threads 

G-CON.4  Operational/Tactical  Threads 

Personnel  familiarity  with  handoff  process 

W-Hum.2_ Task  Understanding 

Information  Processing 

Information  fusion  accuracy 

Proc-Fus.2  Develop  Ship  Folder 

Proc-Crl.1  Multi-Source  Fusion 

Proc-Crl.3  Reference  Data  Fusion 

Information  Sharing 

Content  and  format  interoperability 

DS-Mns.2  Format  for  Distribution 

Boarding 

Communications  reliability 

DS-Dis.1  Optimize  Paths 

DS-Dis.3  Push  Information 


Note  that  the  Program 
Objective  descriptions  in 
Table  5  are  shorthand  for 
the  complete  objective 
statements  in  FIRE. 


Table  5.  Use-case  issues  and  corresponding  MDA  Program  Objectives. 
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6.2  Using  FIRE  for  Planning  Test  Objectives  and  Objective-Goals 

The  first  letters  in  the  codes  shown  in  Table  5  indicates  the  Test  Area  (organized  on  tabs 
in  FIRE2). 

•  DT  =  Detect/Track 

•  Proc  =  Process  Data  and  Infonnation 

•  AD  =  Analyze/Develop  Situation  Understanding 

•  DS  =  Distribute/Share 

•  DB  =  Database  Archive/Retrieve 

•  W  =  Workflow 

•  G  =  Guidance 

•  MIO  =  E-MIO  Operations 

For  definition  of  the  rest  of  the  code  see  Appendix  D. 

Most  of  the  issues  shown  in  Table  5  address  supporting  technologies.  Thus  the  full  Test 
Objective  and  Objective-Goal  statements  address  systems.  However,  there  are  also  two 
Guidance  and  one  Workflow  issues. 

Table  6  contains  the  Test  Objective  and  Objective-Goal  statements  for  each  of  the  issues. 
Note  that  the  technology  objectives  begin  with  the  word  “provide”  or  “develop”.  This  is 
because  the  purpose  of  the  MDA  program  is  to  provide  capabilities.  Objective-Goals  are 
specific  “determinations”  that  assess  the  status  of  the  objective. 

Systems  are  referred  to  as  “system  SSS”  in  Table  6.  For  actual  tests  the  name  of  the 
system  used  would  be  inserted. 

Actual  test  Objective-Goals  would  be  written  more  completely  than 
the  illustrations  presented  here.  Those  shown  here  are  to  illustrate 
the  development  process,  not  for  actual  use. 

Colors  are  used  to  easily  distinguish  a  particular  Test  Objective  and  its  Objective-Goals 
(simply  labeled  Goal  in  the  table).  Test  Objectives  colored  red:  indicate  that  it  was 
decided  during  test  planning  that  they  would  not  be  attempted,  for  reasons  not  given. 

This  is  not  an  unusual  occurrence  during  planning. 


2  At  the  time  of  writing  this  report  the  Program  Objectives  and  Test  Objectives  are  being  formulated  and 
input  to  FIRE.  Thus,  the  below  objectives  statements  may  not  be  exactly  the  same  as  the  final  FIRE 
content. 
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DT-ST.l 

Provide  automatic  tracking  and  reporting  of  non-radiating  Vol  across  the  full  AOR. 

Goal  A  - 

Goal  B  - 

Determine  the  accuracy  and  timeliness  of  system  SSS  Vol  tracking. 

Determine  the  number  of  ships  in  the  AOR  that  system  SSS  can  simultaneously  track. 

DT-ST.2 

Provide  systems  that  automatically  track  and  report  on  radiating  Vol  across  the  full  AOR. 

Goal  - 

Determine  the  persistence,  fraction  of  time,  that  system  SSS  can  track  each  Vol. 

G-TTP.4 

Goal  A  - 

Goal  B  - 

Develop  and  evaluate  TTP  for  MDA  operations. 

Determine  which  Vol  handoff  processes  are  covered  by  TTP 

Determine  which  Vol  handoff  situations  are  covered  by  TTP 

G-CON.4 

Develop  and  evaluate  CONOPS  for  MDA  operations. 

Goal  - 

Determine  which  nation's/unit's  roles/responsibilities  are  covered  by  CONOPS. 

W-Hum.2 

Goal  A  - 

Goal  B  - 

Evaluate  individual  understanding  of  assigned  task  performance. 

Determine  the  amount  of  time  required  for  MOC  personnel  to  accept  Vol  handoff. 

Determine  MOC  personnel  perceived  level  of  understanding  of  handoff  procedures. 

Proc-Crl.l 

Develop  processes  for  fusing  information  from  multiple  intelligence  sources. 

Goal  - 

Determine  which  info  sources  system  SSS  can  automatically  fuse. 

Proc-Fus.2 

Develop  processes  for  correlating  multi-source  ship  information  into  one  assessment. 

Goal  - 

Determine  compatibility  of  each  ship  information  source  with  system  SSS  requirements. 

DS-Mns.2 

Develop  a  means  for  formatting  information  for  distribution  to  multiple  users. 

Goal  A  - 

Goal  B  - 

Determine  the  number  of  formats  that  can  be  automatically  generated  by  system  BBB. 
Determine  accuracy  after  reformatting. 

DS-Dis.2 

Provide  collaboration  capabilities  to  distributed,  disparate  operational  units. 

Goal  A  - 

Goal  B  - 

Goal  C  - 

Determine  the  number  of  users  system  CCC  can  have  in  a  single  collaboration  session. 
Determine  the  number  of  collaboration  services  provided  by  system  CCC. 

Determine  the  bandwidth  required  by  system  CCC. 

Table  6.  Use-Case  Test  Objectives  and  Objective  Goals. 

6.3  Additional  Use-Case  Planning  Components 

The  discussion  above  illustrates  how  objectives  and  goals  are  planned  using  the  NPS 
framework  and  FIRE.  As  noted  in  Section  5.1,  there  is  a  great  deal  more  detailed 
planning  to  be  done.  Some  of  these  MDA  planning  components  (events,  data,  etc.)  are 
already  in  FIRE;  many  are  not.  Even  those  that  are  there  will  probably  need  to  be 
modified  to  be  correct  for  a  specific  MDA  test.  To  illustrate  the  key  components  of 
planning,  we  document  the  planning  for  two  goals  in  Table  7.  All  of  the  input  forms  for 
these  components  are  in  FIRE.  There  are  many  more  planning  components  than  those 
shown  in  Table  7. 
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DT-ST.l  Provide  automatic  tracking  and  reporting  of  non-radiating  Vol  across  the  full  AOR. 

Goal  A  -  Determine  the  accuracy  and  timeliness  of  system  SSS  Vol  tracking. 

Measures 

Accurate:  Report  location  error  (km). 

Timely:  Average  and  maximum  time  between  reports,  all  ships  and  Vol. 

Data  Source 

system  SSS  logs,  ship  navigation  logs. 

Survey 

Is  SSS  reporting  of  ship  positions  sufficient  to  maintain  SA? 

Situations 

System  SSS  in  operation.  Identified  Vol  in  the  AOR. 

Goal  B  -  Determine  the  number  of  ships  in  the  AOR  that  system  SSS  can  simultaneously  track. 

Measures 

Capacity:  Number  of  ship  tracks  displayed  by  system  SSS. 

Capacity:  Number  of  ship  tracks  managed  by  system  SSS. 

Data  Source 

system  SSS  logs,  including  display  log. 

Survey 

How  many  ships  can  be  distinguished  in  the  system  SSS  display? 

Can  the  Vol  be  distinguished  from  other  ships  in  the  display? 

Situations 

System  SSS  in  operation.  Identified  Vol  in  the  AOR. 

Table  7.  Additional  planning  components  for  ship  tracking  test. 
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Appendix  A 

Attribute  Definitions 

Effective  -  Effective  is  an  overarching  attribute.  It  refers  to  how  well  systems,  people, 
and  processes  meet  their  stated  purposes.  This  attribute  has  meaning  only  in  reference  to 
that  purpose.  E.g.,  it  is  not  sufficient  to  state  that  a  system  is  effective  without  also 
stating  at  what. 

Accessible  -  Users  have  access  to  needed  capabilities  and  information.  This  includes 
access  to  communication  means,  data  and  processed  information,  systems,  software, 
support,  etc.  Access  will  often  be  through  a  network.  This  attribute  is  one  of  the  four 
MOE;  its  component  MOP  follow. 

Capacity  -  Number  of  users  that  can  have  access;  number  of  services  that  can  be 
provided;  capacity  of  other  systems  required  for  its  function,  primarily  bandwidth. 
Included  is  information  or  service  throughput. 

Available  -  System  or  capability  is  ready  for  use,  can  be  used,  when  needed.  It  is 
possible  that  a  capability  can  be  accessed  but  cannot  be  used  at  that  time. 

Compatible  -  The  system  or  capability  can  function  with  other  elements  external 
to  it  without  modification  to  either.  It  can  be  integrated  with  other  systems  or 
capabilities.  This  can  also  refer  to  processes  or  organizations  being  compatible  or 
integrated. 

Extensive  -  The  system  or  capability  is  capable  of  servicing  a  large  number  of 
users,  covers  a  large  geographical  area,  services  a  large  number  of  user  types, 
provides  a  number  of  different  types  of  service. 

Efficient  -  The  number  of  steps  or  effort  needed  to  access  and  use  the  service  is 
acceptable.  This  attribute  is  inherently  comparative.  Acceptable  normally  refers 
to  a  standard,  or  an  improvement  over  what  was  formerly  required.  Efficiency 
can  be  a  ratio,  a  judgment  of  (result  obtained)/(effort  required). 

Reliable  -The  capability  or  infonnation  is  there  when  needed,  can  be  depended  on. 
Human  and  organization  reliability  is  included.  This  attribute  is  one  of  the  four  MOE;  its 
component  MOP  follow. 

Robust  -  The  system  or  process  is  able  to  withstand  stress  or  attack.  Changes  in 
environment  are  managed  with  minimal  loss  of  functionality  or  effectiveness. 

Persistent  -  The  system  maintains  its  status  over  long  periods  of  time  (primarily 
ISR  capabilities).  Information  maintains  its  content  and  meaning  across 
processing  and  distribution  means  (e.g.,  tracks). 
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Secure  -  The  system,  process,  information,  has  provisions  that  prevent 
unauthorized  use,  intrusion,  or  tampering. 

Assured  -  Information  is  warranted  to  be  correct,  the  source  identified,  and  non¬ 
repudiation  in  effect.  The  process  is  warranted  to  produce  the  desired  result. 

Capable  -  The  system,  capability,  person,  or  organization  provides  the  needed  services. 
This  attribute  is  one  of  the  four  MOE;  its  component  MOP  follow. 

Sufficient  -  What  has  been  provided/received  is  adequate  for  the  recipient  to 
perform  their  function.  For  humans  and  organizations,  the  skills  available  are 
adequate  for  task  performance.  Sufficiency  can  refer  to  either  quantity  or  level. 

Flexible  -  The  system,  process,  human,  or  organization  responds  easily  to  the 
situation  or  to  changing  requirements.  It  is  adaptable,  can  handle/utilize  a  wide 
range  of  types.  It  is  tailorable/customizable  to  user  needs  and/or  users  can  make 
modifications  to  suite  their  needs. 

Accurate  -  Information  provided  is  correct,  matches  reality  within  acceptable 
limits.  Determinations  of  accuracy  normally  require  definition  of  acceptable  error 
limits. 

Timely  -  The  occurrence  or  delivery  is  within  acceptable  time  limits.  This  can 
refer  to  an  elapsed  time  or  to  meeting  a  schedule. 

Usable  -  The  system,  capability,  infonnation,  or  process  can  be  used.  This  attribute  is 
one  of  the  four  MOE;  its  component  MOP  follow. 

Clear  -  How  the  system  or  process  is  to  be  used  is  easily  understood.  Meaning  of 
the  infonnation  is  easily  comprehended.  Instructions,  guidelines,  definitions  are 
complete  and  meaningful. 

Trusted  -  Users  believe  that  the  infonnation,  process,  system,  organization,  will 
perform  their  function  in  a  manner  that  supports  current  needs. 

Manageable  -  The  system  or  process  can  be  easily  modified  or  manipulated  as 
needs  dictate,  often  in  response  to  changes  in  the  environment.  Included  is 
insuring  that  the  required  level  of  performance  is  maintained.  This  includes 
installation  of  capabilities. 

Relevant  -  Information  provided  applies  to  the  current  situation.  System 
capabilities  are  what  is  needed  for  current  tasks.  Processes  provide  the  actions 
required  for  current  operations. 
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Compliant  -  The  system  or  information  complies  with  standards  or  defined 
structure  and  formats.  Activities  are  in  confonnance  with  existing  CONOPS  and 
TTP. 

Military  Utility  -  Military  utility  is  a  faux  attribute  (not  actually  a  description  of 
characteristics),  used  to  express  that  something  contributes  to  the  perfonnance  of  military 
operations.  It  is  an  overarching  attribute.  The  four  measures  of  utility  follow. 

Improved  -  The  system,  organization,  or  process  improves  the  conduct  of 
military  operations  for  which  they  were  designed. 

Needed  -  The  system,  organization,  or  process  fills  a  gap  an  identified  gap. 

Applicable  -  The  system,  organization,  or  process  is  pertinent  to  conduct  of  the 
operation.  Its  capabilities  match  the  needs  and  conduct  of  the  operation. 

Wanted  -  Operational  personnel  want  the  capability  and  utilize  it.  They  do  not 
currently  have  the  capability  or  would  rather  use  it  in  place  of  other  available 
capabilities. 
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Appendix  B 

MDA  Test  Program  Tracking 

Two  types  of  program  tracking  are  provided: 

•  Objectives  and  attributes  being  evaluated,  by  test  venue 

•  Planning  details  for  each  test  venue 

These  tracking  spreadsheets  are  placed  in  a  workspace  folder  for  easy  access. 

Table  8  shows  a  small  section  of  the  objective  and  attribute  tracking  spreadsheet.  Its 
characteristics  are: 

•  A  sheet  for  each  Test  Area 

•  Program  objective  structure  Attribute  assignments  for  each  Objective  Type 
(indicated  by  an  “X”) 

•  A  row  for  each  venue  (rows  added  as  venues  added) 

•  Attributes  to  be  evaluated  (indicated  by  a  “P”) 

•  Attributes  for  which  results  have  been  obtained  (indicated  by  an  “R”) 

Use  of  the  “P”  and  “R”  indicators  allows  the  spreadsheet  to  be  used  to  track  both 
planning  and  after-test  results  production.  The  Ps  and  Rs  shown  in  the  spreadsheet  are 
notional,  for  illustration,  not  actual  assignments. 

Table  9  shows  an  example  of  the  objective  and  measures  planning  spreadsheet.  There  is 
one  sheet  per  venue.  All  of  the  entries  shown  are  notional,  not  real.  Only  those  MDA 
program  objectives  that  are  planned  to  be  tested  will  be  shown  on  these  sheets. 
Spreadsheet  contents  are: 

•  MDA  Obj  #  Objective  #  from  the  objective  structure. 

•  Objective  Type  Objective  short  title  from  the  objective 

structure. 

•  Workflow  Node  #  Node  #(s)  from  the  workflow  architecture 

that  is/are 

being  exercised  for  this  Test  Goal. 

•  Workflow  Node  Name  Name  of  the  workflow  node. 

•  Operational  Organization  Cmd.  Identification  of  which  Command(s)  will  be 

participating  in  this  objective  test. 

•  Operational  Organization  Cell  Identification  of  the  location(s)/Cell(s) 

within  the 

Command  that  will  be  participating  in  this 
objective 

test  (for  data  capture). 

•  Supporting  Systems  System(s)  that  will  be  used  in  support  of  the 

test 

objective. 
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Test  Objective 
that  will 


Test  Measures 
determined 


Specific  objective  for  that  Objective  Type 

be  tested.  Goals  statements  that  contain  the 
Attributes  are  in  FIRE. 

Specific  Measure(s)  whose  value  will  be 


for  each  Attribute. 


Table  8.  Objective  and  attribute  assessment  planning,  by  venue. 


MDA  Planning  Summary 

Venue:  name  here 

MDA 

Obj  #  Program  Objective 

Workflow 

Node  # 

Node  Name 

Operational  ( 
Command 

Organization 

Cell 

Supporting 

Systems 

Test 

Objective 

Test 

Measures 

DT-SD.2  Radiating-ship 
detection 

WF-26 

ISR  Control 

PACFLT 

zzzz 

CMA 

MASTER 

Determine  the  number 
of  ships  that  can  be 
detected  and  reported. 

Capacity:  Number  of  ships 
reported/hour. 

Accurate:  Reporting  CEP,  nmi. 

DT-ST.4  Handoff 

xxxxx 

yyyyy 

CENTCOM 

PACFLT 

NCIS 

N2 

N2 

CT  Cell 

CMA 

AAA 

Determine  if  handoff  is 
efficient. 

Automatic:  M2M  handoff,  y/n. 
Efficient:  time  required  for  handoff 
(hr);  number  of  steps  required  for  a 
handoff. 

Sufficient:  percent  of  tracking 
information  transmitted. 

CENTRIX 

Capacity:  coalition  throughput. 

Table  9.  Venue  objectives  and  measures  planning. 
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Appendix  C 

Task  /  Attribute  /  Measure  Relationships 

As  noted  in  Section  2.2,  attributes  and  measures  do  not  stand  alone.  They  have  meaning 
only  when  associated  with  an  activity  or  task.  Figure  3  shows  the  various  types  of  task, 
attribute,  and  measure  associations  that  are  encountered  in  military  operations 
assessments.  It  is  useful  to  keep  these  associations  in  mind  when  developing  MDA 
capability  tests.  It  is  not  sufficient  to  test  only  system  performance.  It  is  also  necessary 
to  test  the  processes  that  systems  serve,  and  the  humans  and  organizations  that  execute 
the  processes.  The  relationships  shown  in  Figure  3  illustrate  the  types  of  determinations 
that  should  be  made. 


Figure  3.  Attribute  and  measure  types  and  relationships  to  systems  and  activities. 
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Not  all  associations  can  be  shown  in  the  figure,  e.g.,  collaboration  system  performance  is 
not  shown  associated  with  human  decision  processes. 

The  following  is  an  example  of  task/attribute/measure  interdependence  is  taken  from 
recent  development  work  on  the  NNWC  Capabilities  List  for  Joint  Net-Centric 
Operations. 

Attribute  =  Timely  MOP  =  Timeliness 
Task  =  RFI  response 

Measures  =  a.  Time  from  submission  of  RFI  to  receipt  of  infonnation. 

b.  Time  information  waits  in  queue  for  transmission. 

c.  RFI  processing  time. 

Task  =  network  management 

Measures  =  a.  Time  to  switch  channels. 

b.  Time  from  request  to  receipt  of  access. 

Task  =  infonnation  processing 

Measures  =  a.  Average  time  to  develop  aim-point. 

b.  Fraction  of  mensurated  targets  that  meet  MAAP  cut-off. 
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Appendix  D 

Program  Objectives,  Attributes,  and  Test  Goals 

The  structures,  objectives,  and  metrics  that  have  been  developed  for  MDA  testing  are 
“program  objectives”.  This  means  that  they  contain  the  overall  objectives  and  metrics  of 
the  program.  As  has  been  described  in  this  report,  a  given  test  venue  will  have  a  set  of 
specific  objectives,  goals,  measures,  and  data.  These  specifics  fit  within,  and  are  derived 
from,  the  overall  program  structure. 

Table  10  shows  an  example  of  defined  test  Goals,  attributes  and  measures  for  DT-SD.  1 
and  SD.4,  non-radiating  ship  detection  and  automated  alerting,  respectively  (shown  in 
yellow  in  the  figure).  These  goals  are  actual  assignments  derived  from  the  resource 
documents.  The  thresholds  listed  come  from  the  ONR  S&T  plan,  as  indicated. 

Goals  and  measures  have  been  defined  for  all  of  the  MDA  Program  Objectives.  They  can 
be  found  in  FIRE.  They  can  be  used  directly  for  tests,  or  SMEs  can  design  other  goals 
and  that  fit  within  the  Program  Objectives  structure. 


Goal 

Attribute  Measures 

Threshold  Source 

DT  Detect/Track 

DT-SD  Ship  Detection 

DT-SD.  1  Non-radiating-ship  detection 

Determine  Large  Ship  Detection  Range 

Reach  Detection  Range  (mi) 

2000  mi  S&T 

and  Accuracy 

Accuracy  Location  CEP  (mi) 

5  mi 

DT-SD. 2  Radiating-ship  detection 

DT-SD. 3  Plan  &  Optimize  Collection 

DT-SD. 4  Automated  alerting 

Is  Large  Ship  Detection  alerting  timely? 

Timely  Time  from  detection  to  alert  (hr) 

4  hr  S&T 

Table  10.  Specific  Goals  to  be  evaluated  and  their  attributes  and  measures. 


The  following  tables  list  the  full  MDA  objectives  structure.  Table  1 1  included,  for  each 
Program  Area,  the  Activity/Focus  and  Program  Objectives  (in  condensed  statements). 
Table  12  has  the  principal  attributes  and  brief  indications  of  application  for  each 
Activity/Focus.  These  attributes  apply  to  all  of  the  activity  Program  Objectives  and  the 
specific  ones  to  be  used  are  chosen  during  Test  Objective  and  goal  planning. 

Table  11.  Complete  listing  of  Program  Areas,  Activities  /  Focus,  and  Program  Objectives  (on 
following  pages  28-31). 
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Program  Area  Program  Area 

Activity  /  Focus  Activity  /  Focus 


Program  Objective 

Program  Objective 

DT  Detect/Track 

Proc  Process  Data  &  Information 

DT-SD  Ship  Detection 

Proc-ID  Identify  Ship's  Data 

DT-SD. 1 

Non-radiating-ship  detection 

Proc-ID.  1 

Ship  Characteristics 

DT-SD. 2 

Radiating-ship  detection 

Proc-ID. 2 

Cargo 

DT-SD. 3 

Plan  &  Optimize  Collection 

Proc-ID. 3 

Track  History 

DT-SD. 4 

Automated  alerting 

Proc-ID. 4 

Ship  Infrastructure 

DT-ST  Ship  Tracking 

Proc-ID. 5 

Personnel 

DT-ST.  1 

Non-radiating-ship  tracking 

Proc-ID. 6 

ID  Information  Gaps 

DT-ST.2 

Radiating-ship  tracking 

Proc-Cfy 

Classify  Information 

DT-ST. 3 

AlS-broadcasting-ship  tracking 

Proc-Cfy.  1 

Data  Pedigree 

DT-ST.4 

Handoff 

Proc-Cfy.2 

Data  Currency 

DT-ST. 5 

All  ship  tracking 

Proc-Cfy. 3 

Information  Validity 

DT-Acq 

Acquire  Database  Information 

Proc-Cfy.4 

Assign  Track  ID 

DT-Acq.  1 

Local  repository  retrieval 

Proc-Crl 

Correlate  Information 

DT-Acq. 2 

Remote  repository  retrieval 

Proc-Crl.1 

Multi-Source 

DT-Acq. 3 

Cross-domain  retrieval 

Proc-Crl  .2 

Ship  Information 

DT-Acq  .4 

Weather  &  Environment 

Proc-Crl. 3 

Reference  Data 

DT-Acq.  5 

Reachback/RFI 

Proc-Crl  .4 

Intelligence  information 

DT-Int  Intelligence  Collection 

Proc-Crl.  5 

Track  Information 

DT-Int. 1 

Collect  Requirements 

Proc-Fus 

Fuse  Information 

DT-lnt.2 

Research  RFI 

Proc-Fus. 1 

Format  Information 

DT-Int.  3 

Validate  &  Prioritize  Req. 

Proc-Fus. 2 

Develop  Total  Ship  Folder 

DT-lnt.4 

Synchronize  Collection  Req. 

Proc-Fus. 3 

Attach  Meta-Data 

DT-Int.  5 

Simulate  ISR  Plan 

DT-Int.  6 

Collection  Plan 

DT-Int. 7 

HUMINT 

DT-Int.  8 

Alerting  /  l&W 

DT-Int.  9 

PED 
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Program  Area 

Activity  /  Focus 

Program  Objective 


Analyze/Develop  SA 

Develop  Profiles 

AD-Prof.1 

Develop  Behavior  Profiles 

AD-Prof.2 

Develop  Threat  Profiles 

AD-Prof.3 

Correlate  With  Ship  Information 

Request  Information 

AD-Req.1 

Prioritize  Information  Gaps 

AD-Req.2 

Request  ISR  Collection 

AD-Req.3 

Request  Database  Information 

AD-Req.4 

Request  HUMINT 

Classify  Vessels 

AD-Cfy.1 

Classify  Ships 

AD-Cfy.2 

Anomaly  Detection 

AD-Cfy.3 

Identify  Vol 

AD-Cfy.4 

Classify  Vol 

Prioritize  Information 

AD-Pri.1 

Prioritize  Information 

AD-Pri.2 

Prioritize  Distribution 

Program  Area 

Activity  /  Focus 

Program  Objective 

Distribute/Share 


Distribution  Means 

DS-Mns.1 

Select  Distribution  Means 

DS-Mns.2 

Format  for  Distribution 

DS-Mns.3 

Select  /  Authorize  Recipients 

Collaborate 

DS-Coll.1 

Authorize  Participants 

DS-Coll.2 

Prepare  Information 

DS-Coll.3 

Display  Information 

DS-Coll.4 

Manage  Collaboration 

Disseminate 

DS-Dis.1 

Optimize  Distribution  Paths 

DS-Dis.2 

Update  Databases 

DS-Dis.3 

Push  Information 

DS-Dis.4 

Alert  Recipients 

Program  Area  Program  Area 

Activity  /  Focus  Activity  /  Focus 

Program  Objective  Program  Objective 


Database  Archive/Retrieve 

Deploy  Databases 

Guidance 

TTP  &  SOP 

DB-Dep.1 

Local  Repository 

G-TTP.1 

Distribution 

DB-Dep.2 

Central  Repository 

G-TTP.2 

Operations  Coverage 

DB-Dep.3 

Multi-Level  Guards 

G-TTP.3 

Situation  Coverage 

DB-Dep.4 

Cross-Domain  Repositories 

G-TTP.4 

Operational/Tactical  Threads 

Acquire  Information 

G-TTP.5 

Technology  Inclusion 

DB-Acq.1 

Human 

CONOPS 

DB-Acq.2 

Ship 

G-CPS.1 

Command  Relationships 

DB-Acq.3 

Ship  Status  /  Tracks 

G-CPS.2 

Operations  Coverage 

DB-Acq.4 

Environment 

G-CPS.3 

Situation  Coverage 

DB-Acq.5 

Cargo 

G-CPS.4 

Operational/Tactical  Threads 

DB-Acq.6 

Intelligence 

G-CPS.5 

Technology  Inclusion 
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DB-Ass.1  Monitor  Repository  Status 
DB-Ass.2  Protect  Repositories 

DB-Ass.3  Detect  Unauthorized  Access 

DB-Ass.4  Detect  Information  Defects 

DB-Ass.5  Status  Alerting 

DB-Ass.6  Failover 
DB-Ass.7  Repair  Defects 


Authenticate  Information 

G-CPS.6  Quality  of  Instructions 

DB-Ath.1  Authenticate  Source 

Standing  Orders 

DB-Ath.2  Assure  Source 

G-SO.1  Distribution 

DB-Ath.3  Attach  Metadata 

G-SO.2  Match  to  Situation 

DB-Ath.4  Assure  Information  Quality 

G-SO.3  Updating 

Manage  Access 

G-SO.4  Quality  of  Orders 

DB-Acc.1  Authorize  Users 

Cross-Organization  Agreements 

DB-Acc.2  Profile  Users 

G-CO.1  Multi-Command 

DB-Acc.3  Provide/Control  Access 

G-CO.2  Multi-Department 

Assure  Information 

G-CO.3  Multi-National 

Program  Area 

Activity  /  Focus 

Program  Objective 


Workflow 

Task  Assignments 

W-Arch.1 

Task  Organization 

W-Arch.2 

Information  Flow 

W-Arch.3 

Prioritization 

W-Arch.4 

Workflow 

Organization 

W-Org.1 

Organization  Structure 

W-Org.2 

Command  Relationships 

W-Org.3 

Organization  Dynamics 

W-Org.4 

Situation/Organization  Match 

W-Org.5 

Communications  Structure 

W-Org.6 

Adaptation 

W-Org.7 

Performance 

Group 

W-Grp.1 

Group  Competence 

W-Grp.2 

Activities  Understanding 

W-Grp.3 

Situation/Group  Structure  Match 

W-Grp.4 

Performance 

Program  Area 

Activity  /  Focus 

Program  Objective 


E-MIO  Operations 

Information  Acquisition 

MIO-Acq.1 

Vessel  Characteristics 

MIO-Acq.2 

Threat  Assessment 

MIO-Acq.3 

Rules  and  Orders 

MIO-Acq.4 

Available  Assets  Status 

MIO-Acq.5 

Formulate/issue  RFIs 

MIO-Acq.6 

Environment/Weather 

Develop  Situation  Understanding 

MIO-SU.1 

Correlate  Information 

MIO-SU.2 

Track/Vol  Status 

MIO-SU.3 

Assess  Tactical  Environ. 

MIO-SU.4 

Assess  Hazards 

MIO-SU.5 

Assess  Urgency 

MIO-SU.6 

Infer  Vol  Intent 

Course-of-Action 

MIO-CoA.1 

Collaborate,  Develop  CoA 

MIO-CoA.2 

Simulate  CoA 

MIO-CoA.3 

Present  CoA 
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W-Grp.5 

Capabilities/Task  Match 

W-Grp.6 

Adaptation 

W-Grp.7 

Workload 

Human 

W-Hum.1 

Competence 

W-Hum.2 

Task  Understanding 

W-Hum.3 

Performance 

W-Hum.4 

Capabilities/Task  Match 

W-Hum.5 

Adaptation 

W-Hum.6 

Workload 

Multi-Organization  Workflow 

W-MOrg.1 

Organization  Structure 

W-MOrg.2 

Command  Relationships 

W-MOrg.3 

Organization  Dynamics 

W-MOrg.4 

Situation/Organization  Match 

W-MOrg.5 

Communications  Structure 

W-MOrg.6 

Adaptation 

W-MOrg.7 

Performance 

MIO-CoA.4 

Select  COA 

MIO-CoA.5 

Develop  Tasking 

MIO-CoA.6 

Develop  Safety  Plan 

MIO-CoA.7 

Disseminate  Tasking 

Boarding  Execution 

MIO-Brd.1 

Insure  Safety 

MIO-Brd.2 

Execute  Boarding 

MIO-Brd.3 

Direct  Forces 

MIO-Brd.4 

Cross-Domain  Collaboration 

MIO-Brd.5 

Real-Time  Reporting 

MIO-Brd.6 

Visual/TV  Monitoring 

MIO-Brd.7 

Mission  Reports 

MIO-Brd.8 

Dynamic  Re-Tasking 

Shipboard  Collection 

MIO-Coli.1 

CBNRM 

MIO-Coll.2 

Biometrics 

MIO-Coll.3 

Ship  Information 

MIO-Coli.4 

Cargo 

MIO-Coll.5 

Video 

Information  Dissemination 

MIO-Dis.1 

RHIB  Relay 

MIO-Dis.2 

LOS 

MIO-Dis.3 

VOI  Internal 

MIO-Dis.4 

SATCOM 

MIO-Dis.5 

OTH 

Mission  Assessment 

MIO-Ass.1 

Real-Time  Assessment 

MIO-Ass.2 

Real-Time  Feedback 

EM-Ass.3 

After-Action  Assessment 

EM-Ass.4 

Database  Update 

EM-Ass.5 

Assessment  Dissemination 

Table  12  contains  principal  attributes  for  MDA  activities  and  test  foci.  For  each  attribute 
there  is  included  an  application  which,  indicates  the  type  of  measure  that  is  to  be 
produced.  Neither  the  attributes  nor  their  application  is  complete;  others  may  be  used  in 
developing  a  test.  Application  is  shown  rather  than  an  actual  measure  for  simplicity, 
showing  the  focus. 

Table  12.  Program  Area  Activity  /  Focus  and  their  Attributes  and  attribute  applications  (on 
following  pages  33-37). 
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Program  Area 

Activity  /  Focus 

Attribute  Application 


DT  Detect/Track 

DT-SD  Ship  Detection 

Extensive 

area  covered,  %  of  AOR 

Accurate 

location  and  ID 

Timely 

reporting  delays,  alerts 

Efficient 

collection  planning 

Persistent 

collection,  monitoring 

Capacity 

simultaneous  #  of  ships 

Deployable 

detection  system 

Reach 

detection  range 

DT-ST  Ship  Tracking 

Accessible 

area  to  be  monitored 

Available 

information  to  users 

Persistent 

tracking 

Efficient 

handoff 

Accurate 

location  and  ID 

Timely 

reporting  delay 

Automatic 

tracking  and  alerting 

DT-Acq  Acquire  Database  Information 

Accessible 

repository  to  users 

compatible 

information  formats,  systems 

Available 

repository  information 

Robust 

against  penetration,  corruption 

Capacity 

information  throughput 

Reliable 

information  availability 

DT-Int  Intelligence  Collection 

Accurate 

data/information  collected 

Sufficient 

ISR  resources 

Timely 

RFI  response,  l&W  alerts 

Trusted 

data  sources 

Assured 

data  pedigree 

Compliant 

priorities  to  requirements 

Relevant 

collection  plan  to  requirements 

Available 

information  sources 

Persistent 

collection 

Program  Area 

Activity  /  Focus 

Attribute  Application 


Proc  Process  Data  &  Information 

Proc-ID  Identify  Ship's  Data 

Capacity 

#  of  tracks,  data  amount 

Accurate 

ship/data  association 

Automatic 

data  entry,  categorization 

Assured 

data  pedigree 

Proc-Cfy 

Classify  Information 

Efficient 

automation,  personnel  savings 

Timely 

time  to  classify 

Assured 

information  pedigree 

Accurate 

classification,  track  number 

Automatic 

classification 

Proc-Crl 

Correlate  Information 

Capacity 

number  of  ships,  sources 

Compatible 

data  formats,  systems 

Flexible 

number  of  source  types 

Assured 

data  pedigree,  source 

Automatic 

ID,  correlation 

Timely 

time  per  ship 

Proc-Fus 

Fuse  Information 

Capacity 

number  of  ships,  sources 

Compatible 

data  formats,  systems 

Flexible 

number  of  source  types 

Assured 

data  pedigree,  source 

Automatic 

fusion 

Timely 

time  per  ship 

Accurate 

ship  association 
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Program  Area  Program  Area 

Activity  /  Focus  Activity  /  Focus 

Attribute  Application  Attribute  Application 


Analyze/Develop  SA 

Develop  Profiles 

Distribute/Share 

Distribution  Means 

Automatic 

profile  generation,  correlation 

Compatible 

data,  systems,  protocols 

Automatic 

Anomaly  Detection 

Deployable 

to  platforms,  units 

Timely 

alerts 

Capacity 

bandwidth,  #  paths,  throughput 

Flexible 

number  of  profile  types 

Manageable 

auto-failover,  status  reports 

Accurate 

profiles,  threat  ID 

Automatic 

prioritization,  means,  distribution 

Clear 

profile  structure,  relationships 

Accurate 

data  drops,  jitter 

AD-Prof.3 

Correlate  With  Ship  Information 

Collaborate 

Request  Information 

Available 

collaboration  toolkit 

Relevant 

information  gap  ID 

Reliable 

tools  availability,  functionality 

Available 

information  sources 

Flexible 

presentation  means 

Timely 

RFI  development,  submission 

Timely 

time  to  join,  establish  session 

Compliant 

request  format 

Reach 

number  of  users 

Classify  Vessels 

Manageable 

reconfigurable,  user  &  status  rpts. 

Accessible 

reference  data,  including  M2M 

Disseminate 

Capacity 

#  of  classifications,  data  amount 

Timely 

transmission  time,  alerts 

Automatic 

classification,  Vol  ID,  prioritization 

Available 

selected  transmission  option 

Available 

required  data  sources 

Reach 

distribution  area,  #  of  customers 

Prioritize  Information 

Robust 

automatic  redirect,  jamming 

Sufficient 

information  for  prioritization 

Automatic 

distribution  recipient,  alerts 

Relevant 

priorities  to  situation 

Persistent 

network  available 

Flexible 

#  of  priority  types 

Flexible 

#  of  distribution  options 

Compliant 

priorities  with  orders,  situation 

Assured 

delivery  receipt 

Capacity 

#  of  tracks  prioritized 

35 


Program  Area 
Activity  /  Focus 

Attribute  Application 


Database  Archive/Retrieve 

Deploy  Databases 

Capacity 

storage,  information  types 

Manageable 

automated  recovery,  status  reports 

Compatible 

systems,  protocols  interoperability 

Acquire  Information 

Automatic 

data  pull,  archiving 

Available 

external  data,  data  to  users 

Accessible 

user  to  repository,  data  pull 

Compatible 

systems,  formats 

Flexible 

source  types,  data  types.  Formats 

Authenticate  Information 

Assured 

source  logging,  pedigree,  marking 

Compliant 

with  standards 

Manage  Access 

Extensive 

#  of  users,  profiles  managed 

Flexible 

#  of  user  types,  networks 

Secure 

#  unauthorized  uses 

Persistent 

access,  down  time 

Manageable 

set-up  efficiency,  profiles 

Assure  Information 

Extensive 

IA  across  domains 

Robust 

backup,  failover,  down  time 

Sufficient 

IA  processes  and  systems 

Available 

repository  monitoring 

Manageable 

status  reports  accuracy,  access 

Timely 

defect  repair 

Automatic 

status  reporting  and  alerting 

Program  Area 

Activity  /  Focus 

Attribute  Application 


Guidance 

TTP  &  SOP 

Available 

promulgated  and  on  hand 

Compatible 

across  nations  and  units 

Compliant 

with  doctrine  &  directives 

Flexible 

alternate  actions  described 

Trusted 

outcomes  produced 

Relevant 

applies  to  the  situation 

Clear 

directions 

CONOPS 

Available 

promulgated  and  on  hand 

Compatible 

across  nations  and  units 

Compliant 

with  doctrine  &  directives 

Flexible 

alternate  actions  described 

Trusted 

outcomes  produced 

Relevant 

applies  to  the  situation 

Clear 

directions 

Standing  Orders 

Available 

promulgated  and  on  hand 

Compatible 

across  nations  and  units 

Compliant 

with  doctrine  &  directives 

Flexible 

alternate  actions  described 

Trusted 

outcomes  produced 

Relevant 

applies  to  the  situation 

Clear 

directions 

Cross-Organization  Agreements 

Compatible 

national  doctrine,  CONOPS 

Available 

in  existence 

Clear 

understood  at  all  levels 

Extensive 

#  of  situations 

Sufficient 

coverage  of  situation 
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Program  Area 
Activity  /  Focus 

Attribute  Application 


Workflow 

Task  Assignments 

Sufficient 

task  coverage,  information 
products 

Efficient 

task  performance,  work  required 

Clear 

personnel  understand 

Timely 

information  for  task 

Flexible 

to  changing  technologies, 
situations 

Compatible 

across  MOCs  and  units 

Organization 

Efficient 

task  performance,  work  required 

Compatible 

across  MOCs,  units,  and  nations 

Sufficient 

covers  requirements,  needs 

Flexible 

to  changing  CONOPS,  agreements 

Timely 

information  delivery 

Trusted 

decision  processes 

Group 

Flexible 

to  changing  missions 

Manageable 

for  personnel  turnover 

Compatible 

skills  match  tasks 

Capacity 

to  handle  workload 

Sufficient 

training  to  achieve  competence 

Deployable 

for  command  portability 

Human 

Flexible 

to  task  assignments,  workflow 

Sufficient 

training  to  achieve  competence 

Compatible 

skills  to  task  match 

Robust 

to  changing  situation 

Capable 

task  performance 

Accurate 

task  performance 

Capacity 

workload 

Multi-Organization  Workflow 

Flexible 

to  different  missions,  situations 

Clear 

cross-organization  task  handoff 

Trusted 

cross-organization  products 

Compatible 

cross-organization  workflow 

Compatible 

cross-organization  products 

Program  Area 

Activity  /  Focus 

Attribute  Application 


E-MIO  Operations 

Information  Acquisition 

Available 

needed  info  and  databases 

Sufficient 

information  for  assessments 

Relevant 

information  applies  to 
situation 

Accurate 

information 

Timely 

information  updates, 
freshness 

Develop  Situation  Understanding 

Capacity 

number  of  ships  evaluated 

Efficient 

time  required  per  ship 

Flexible 

information  types,  ship  types 

Sufficient 

SU  for  CoA  development 

Relevant 

evaluation  to  situation 

Compatible 

with  command  priority 

Course-of-Action 

Efficient 

CoA  development  time 

Flexible 

CoA  considered,  options 

Relevant 

to  situation,  command  priority 

Compliant 

with  command  priority 

Sufficient 

information  for  CoA  decision 

Timely 

in  time  for  execution 

Clear 

CoA  options,  actions 

Boarding  Execution 

Clear 

tasking,  execution  reports 

Timely 

execution  of  tasking 

Accurate 

taking  execution 

Flexible 

immediate  situation  response 

Deployable 

execution  forces 

Sufficient 

forces  for  execution 

Reliable 

execution  reporting 

Compliant 

execution  with  directions 

Shipboard  Collection 

Accessible 

ship,  personnel,  manifests 

Capacity 

total  biometrics 

Accurate 

data  collection 

Timely 

data  collection 
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Trusted 

Sufficient 

data  collection  devices 

data  collection  for  threat  eval 

Information  Dissemination 

Accessible 

comms  paths,  internal  & 
external 

Capacity 

throughput 

Timely 

data  receipt 

Manageable 

communications  architecture 

Secure 

communications  paths 

Flexible 

path  switching,  data  formats 

Mission  Assessment 

Sufficient 

assessment  for  decisions 

Clear 

assessment  for  decisions 

Timely 

in  time  for  re-tasking 

Relevant 

to  situation,  priorities,  intent 

Accurate 

relation  to  ground  truth 
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Appendix  E 

Mapping  to  JCIDS  JCAs 


Mapping  MDA  activities  to  the  JCIDS  JCAs  is  not  part  of  this  project.  However,  it  may 
be  useful  at  some  point  to  do  so.  The  purpose  of  this  appendix  is  to  illustrate  the  method 
that  has  been  developed  to  do  such  mapping. 

Structures  and  methods  have  been  developed  for  NAVNETWARCOM  to  map  their 
capabilities  lists  to  the  JCIDS  JCAs.  (See  “Mapping  Experiment  Results  to  Operational 
Capabilities”,  NPS-97-08-001,  Nov  2007).  Mapping  is  done  through  correlation  of 
operational  activities  using  an  Operational  Activities  Set.  The  MDA  program  objectives 
presented  in  this  report  are  consistent  with  the  Operational  Activities  Set. 

E.l  Operational  Activity  Set 

The  JCAs  include  supporting  and  supported  activities.  The  supported  activities  structure 
follows  the  classic  OODA  loop.  The  supporting  activities  are  generally  accepted  to  be 
Net-Centric  Operations,  Battlespace  Awareness,  and  C2.  In  addition  to  OODA,  they 
have  their  own  structure,  designated  “Service”,  to  include  such  things  as  network 
installation. 

The  Operational  Activity  Set  has  a  3-level  structure: 

•  Level- 1,  Operational  Area  (e.g.,  land  operations,  surface  warfare,  battlespace 
awareness). 

•  Level-2,  Activity  Category 

o  Observe 
o  Orient 
o  Decide 
o  Act 
o  Service 

•  Level-3  contains  Activity  Types  under  each  of  these  Categories 

•  Level-4  contains  Tasks  under  each  Activity  Type 

Table  13  lists  the  Activity  Categories  and  Types.  The  chronological  view  of  activities 
presented  in  Table  13  is  the  most  intuitive  and  illustrates  information  development.  For 
example,  the  table  shows  that  data  is  processed  into  information  in  the  Observe  phase, 
distributed,  then  acquired  and  processed  into  SA  in  the  Orient  phase  (note  the  overlaps  of 
distribute  and  acquire  to  illustrate  their  overlap  and  interdependence). 
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Observe 

Orient 

Decide 

Act 

Service 

Ob-Plan 

Ob- 

AcqD 

Continuous 

1 

Ob- 

ProcD 

S-Plan 

Ob-Disl 

Or-Acql 

1 

D  =  Data 

Or-Procl 

S-Acquire 

1  =  Information 

Or-DevSA 

1 

K  =  Knowledge 

Or-ShrSA 

S-Manage 

SA  =  Sit.  Aware. 

Or-PntSA 

D-AcqK 

1 

SU  =  Sit.  Under. 

Or-Guide 

D-DevSU 

S-Assure 

T  =Tasking 

D-ShrSU 

1 

Mon  =  Monitor 

D-DevCoA 

S-Authorize 

Rprt  =  Report 

D-PntCoA 

1 

Ex  =  Execute 

D-CoA 

S-Distribute 

Acq  =  Acquire 

D-DevT 

1 

Proc  =  Process 

D-DisT 

A-AcqT 

S- In  struct 

Dis  =  Distribute 

A-DisUT 

1 

Shr  =  Share 

A-Ex 

1 

Dev  =  Develop 

A-ExMon 

1 

UT  =  Unit  Tasking 

A-ExRprt 

| 

Guide  =  Guidance 

Table  13.  Chronological  Operational  Activities  Category  and  Type. 

Table  14  presents  a  view  of  the  set  that  illustrates  the  commonality  of  activities  across  the 
OODA  categories. 


Activity 

Type 

Observe 

Cate 

Orient 

gory 

Decide 

Act 

Service 

Plan 

Ob-Plan 

Acquire 

Ob-AcqD 

Or-Acql 

D-AcqK 

A-AcqT 

Plan 

Process 

Ob-ProcD 

Or-Procl 

Develop 

Or-DevSA 

D-DevSU 

D-DevCoA 

Acquire 

Distribute 

Ob-Disl 

Or-ShrSA 

D-DisT 

A-DisT 

Manage 

Assure 

Present 

Or-PntSA 

D-PntSU 

D-PntCoA 

Execute 

A-Ex 

A-ExMon 

A-ExRprt 

Authorize 

Guidance 

Or-Guide 

D-CoA 

Distribute 

D-DevT 
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|  jlnstruct 


Table  14.  Operational  Activity  Set  sorted  by  Activity-Type. 

A  complete  description  of  the  Operational  Activity  set  is  found  in  the  NPS  report: 
“Mapping  Experiment  Results  to  Operational  Capabilities”,  NPS-97-08-001,  Nov  2007. 


E.2  Mapping  MDA  Test  Results 

Mapping  MDA  test  to  the  MDA  program  is,  of  course  direct.  Mapping  to  other  areas  is 
done  via  the  Operational  Activity  Set  and  the  mapping  of  that  Set  to  those  areas.  The 
process  is  straightforward  and  Figure  4  illustrates  its  rudiments. 


Figure  4.  Mapping  MDA  results  to  MDA  program  objectives  and  to  other  areas. 
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5.  MDA  in  the  FIRE  Knowledge  Management  System 

NPS  has  developed  the  FORCEnet  Innovation  and  Research  Enterprise  (FIRE)  to  provide 
complete  support  for  experiment  and  test  planning  and  reporting.  An  MDA  section  has  been  sel 
up  in  FIRE,  implementing  the  structure  described  above.  In  this  chapter,  we  describe  the 
system’s  capabilities  and  how  it  is  used. 

It  is  important  to  recognize  that  the  objective  statements  in  FIRE  are  Test  Objectives. 

They  are  the  actual  objectives  to  be  achieved  and  their  status  addressed  in  a  test  venue 

5.1  FIRE  Structure 

FIRE  has  two  basic  sections,  Planning  Forms  and  Workspace: 


Planning  Forms 

•  Detailed  planning  is  accomplished  through  entries  in  pre-set  forms. 

•  There  are  three  sets  of  planning  forms 

o  Objective 
o  Data/Events 
o  Results 

•  Each  forms  set  includes 

o  Input/Edit  form 

o  Report  that  shows  the  current  planning  database  content 

Workspace 

•  Contains  collaboration  capabilities  and  folders  for  information  to  be  shared 

•  Pre-created  folders  and  user  defined  folders 

•  Check-in/check-out  library  for  document  version  control 


Following  is  a  depiction  of  the  principal  components  of  the  planning  forms. 


Test  Objective  Planning 


Test  Objective 


Objective-Goal 


Attributes 


Measures 


Surveys 


Data/Event  Planning 

Events 


Situations 


Data  Specifics 


Collection  Procedures 


Collector  Instructions 


Archiving  Procedures 


Thread  Results 


Measures  Results 


Survey  Answers 


Test  Context 


Obj-Goal  Status 


Context  Impact 


Figure  1.  Principal  components  of  the  MDA  planning  forms  in  FIRE. 
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